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Listing of Claims: 

1. (Currently Amended) A computer assisted/implemented method for 
developing a classifier for classifying electronic communications comprising the steps of: 

(a) presenting electronic communications to a user for labeling as relevant or 
irrelevant, the communications being selected from groups of electronic communications 
including: 

a training set group of communications, the training set group of 
communications being selected by a traditional an active learning algorithm; 

a system-labeled set of communications previously labeled by the system; 

a test set group of communications, die test set group of communications for 
testing the accuracy of a current state of fee a classifier being developed by the 
present method ; 

a faulty set of communications suspected to be previously mis-labeled by the 
user; and 

a random set of communications previously labeled by the user; 

(b) developing [[a]] the classifier for classifying communications based upon the 
relevant/irrelevant labels assigned by the user during the presenting step of the electronic 
communications to the user; and 

(c) deploving the classifier for use in classifving electronic communications based 
upon die relevant/irrelevant labels . 
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2. (Currently Amended) The method of claim 1, wherein the presenting step of 
the electronic communications to the user includes the steps of : 

assessing a value related to performance that labeling a set of communications from 
each group will provide provides to the classifier being developed; and 

selecting a next group for labeling based upon ^ a greatest respective value that will 
be provided to the classifier being developed from the assessing step. 

3. (Currently Amended) A computer assisted/implemented method for 
developing a classifier for classifying electronic communications (text, electronic, etc.) 
comprising the steps of: 

(a) presenting communications to a user for labeling as relevant or irrelevant, the 
communications being selected from groups of communications including: 

a training set group of communications, the training set group of 
communications being selected by a traditional m active learning algorithm; 

a test set group of communications, the test set group of communications for 
testing the accuracy of a current state of fee a classifier being developed by the 
present method ; and 

a previously-labeled set of communications previously labeled by at least one 
of the user, the system and another user; 

(b) developing [[a]] die classifier for classifying communications based upon the 
relevant/irrelevant labels assigned by the user during the presenting step : and 
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(c) deploying the classifier for use in classifying electronic communications based 
upon the relevant/irrelevant labels . 

4. (Original) The method of claim 3, wherein the preyiously-labeled set of 
communications includes communications previously labeled by the user. 

5. (Currently Amended) The method of claim 4, wherein the previously-labeled 
set of communications includes communications suspected by the system to be possibly mis- 
labeled by the user. 

6. (Original) The method of claim 3, wherein the previously- labeled set of 
communications includes communications previously labeled by the system. 

7. (Original) The method of claim 3, wherein the previously-labeled set of 
communications includes communications previously labeled by a user and communications 
previously labeled by the system. 

8. (Currently Amended) The method of claim 3 wherein fee presenting st^ the 
electronic communications to the user includes the steps of : 

assessing a value that labeling a set of communications from each group will provide 
to the classifier being developed; and 

selecting a next group for labeling based upon the greatest respective value that will 
be provided to the classifier being developed from the assessing step. 

9. (Currently Amended) The method of claim 3 wherein fee presenting stsp the 
electronic communications to fee user includes fee steps of : 
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assessing a value that labeling a set of communications from each group will provide 
to the classifier being developed; and 

selecting a next group for labeling based upon ^ achieving known performance 
bounds for the classifier. 

10. (Currently Amended) The method of claim 3 further comprising the step of 
developing an expression of labeling criteria in an interactive session with the user. 

11. (Currently Amended) The method of claim 10, wherein the interactive session 
includes the steps of p osing hypothetical questions to the user regarding what type of 
information the user would consider relevant. 

12. (Original) The method of claim 11, wherein the hypothetical questions 
elicit "yes", "no" and "unsure" responses from the user. 

13. (Original) The method of claim 1 1 wherein subsequent questions are 
based, at least in part, upon the answers given to previous questions. 

14. (Currently Amended) The method of claim 1 1 wherein the step of developing 
an expression of labeling criteria produces a criteria document. 

15. (Original) The method of claim 14 wherein the expression and/or the 
criteria document include a group of keywords and/or phrases for use by the system in 
automatically labeling communications. 

16. (Currently Amended) The method of claim 10 wherein the step of developing 
an expression of labeling criteria produces a criteria document. 
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17. (Currently Amended) The method of claim 16 wherein the criteria document 
includes a list of items that are considered relevant and a list of things items that are 
considered irrelevant. 

18. (Currently Amended) The method of claim 17, wherein to presenting step (a) 

the electronic communications to the user includes the step of querying the user which 
item(s) influenced the label on a user-labeled communication. 

19. (Currently Amended) The method of claim 40 16 wherein the expression 
and/or the criteria document include a group of keywords and/or phrases for use by the 
system in automatically labeling communications. 

20. (Currently Amended) The method of claims claim 10 wherein the interactive 
session is conducted prior to to presenting the electronic communications to the user st^ 
(a). 

21 . (Currently Amended) A computer assisted/implemented method for 
developing a classifier for classifying electronic communications (text, electronic, etc.) 
comprising the steps of : 

(a) developing an expression of labeling criteria in an interactive session with to a 

user; 

(b) presenting communications to [[a]] die user for labeling as relevant or irrelevant; 

(c) developing a classifier for classifying communications based upon the 
relevant/irrelevant labels assigned by the user during the presenting step ; and 
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(d) deploying the classifier for use in classifying electronic communications based 
upon the relevant/irrelevant labels : 

wherein at least one of th e pr e s e nting st e p (b) and th e d e v e loping st e p (c) use the 
expression of labeling criteria developed in the developing step (a). 

22. (Currently Amended) The method of claim 21, wherein the interactive session 
includes the steps of posing questions to the user regarding what type of information the user 
would consider relevant. 

23. (Original) The method of claim 22, wherein the questions elicit "yes", 
"no" and "unsure" responses from the user. 

24. (Currently Amended) The method of claim 21 22 wherein subsequent 
questions are based, at least in part, upon the answers given to previous questions. 

25. (Currently Amended) The method of claim 21 22 wherein the questions are 

structured from several dimensional levels of relevance, including a first dimension of 
question segments on a topic, a second dimension of question segments on an aspect of the 
topic and a third dimension of question segments on a type of discussion. 

26. (Original) The method of claim 25, wherein: 

the first dimension of question segments on a topic include one or more of the 
following segments: a segment concerning a client's product and a segment concerning a 
client's competitors; 
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the second dimension of question segments on a topic include one or more of the 
following segments: a segment concerning a feature of the first segment, a segment 
concerning the first segment itself, a segment concerning corporate activity of the first 

segment, a segment concerning price of the first segment, a segment concerning news of the 
first segment and a segment concerning advertising of the first segment; 

the third dimension of question segments on a topic include one or more of the 
following segments: a segment concerning a mention of the second dimension segment, a 
segment concerning a description of the second dimension segment, a segment concerning a 
usage statement about the second dimension segment, a segment concerning a brand 
comparison involving the second dimension segment, and a segment concerning an opinion 
about the second dimension segment. 

27. (Currently Amended) The method of claim 21 wherein the step of developing 
m flie expression of labeling criteria produces a criteria document. 

28. (Original) The method of claim 27 wherein the criteria document includes 
a list of items that are considered relevant and a list of items that are considered irrelevant. 

29. (Original) The method of claim 28 wherein the criteria document includes 
a group of keywords for use by the system in automatically labeling communications. 

30. (Currently Amended) The method of claim 28, wherein the presenting step (b) 
the electronic conmiunications to the user includes th e st e p of querying the user which items 
influenced the label on a user-labeled conmiunication. 
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3 1 . (Original) The method of claim 2 1 wherein the expression of labeling 
criteria includes a group of keywords and/or phrases for use by the system in automatically 
labeling communications. 

32. (Currently Amended) The method of claim 31 wherein the group of keywords 
is also for use by the system in a step of gathering communications. 

33. (Currently Amended) A computer assisted/implemented method for 
developing a classifier for classifying electronic communications (text, electronic, etc.) 
comprising the steps of : 

(a) defining a domain of communications on which a ^ classifier is going to operate; 

(b) collecting a set of communications from the domain; 

(c) eliciting labeling communication criteria from a user; 

(d) labeling, by the system, communications from the set of communications 
according, at least in part, to the labeling communication criteria elicited from the user; 

(e) labeling, by the user, communications from the set of communications; 

(f) building flie [[a]] communications classifier according to a combination of labels 
applied to communications in labeling steps (d) and (e) ; and 

(g) deploying the classifier for use in classifying electronic communications based 
upon the combination of labels . 

34. (Currently Amended) The computer implemented method of claim 33, 
wherein the combination of the labeling steps (d) and (e), and the building step (f) includes 
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the step of selecting communications for labeling by the user targeted to build the 
communications classifier within known performance bounds. 

35. (Currently Amended) The computer implemented method of claim 34, 
wherein fee selecting st^ communications for labeling by the user selects communications 
from groups of communications including: 

a training set group of communications, the training set group of 
communications-being selected by a traditional an active learning algorithm; 

a test set group of communications, test set group of communications for testing the 
accuracy of a current state of the classifier being developed by the present method ; and 

a previously-labeled set of communications previously labeled by at least one of the 
user, the system and another user. 

36. (Currently Amended) The computer implemented method of claim 34, 
wherein the selecting step communications for labeling by the user selects communications 
from groups of communications including: 

a training set group of communications , the training set group of communications 
being selected by a traditional m active learning algorithm; 

a system-labeled set of communications previously labeled by the system; 

a test set group of communications, test set group of communications for testing the 
accuracy of a current state of the classifier being developed by the present method ; 
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a faulty set of communications suspected to be previously mis-labeled by the user; 

and 

a random set of communications previously labeled by the user. 

37. (Currently Amended) The computer implemented method of claim 33, 
wherein the communication criteria elicited in the eliciting st^ (c) is used, in part, to 
determine communications to collect in the collecting step (b). 

38. (Currently Amended) The computer implemented method of claim 37, 
wherein the eliciting step (c) involves an interactive session with the user. 

39. (Currently Amended) The computer implemented method of claims claim 37, 
wherein the communication criteria elicited in the eliciting st^ (c) is used, in part, by the 
system to label communications in the labeling st^ (d). 

40. (Currently Amended) The computer implemented method of claim 39, 
wherein the eliciting st^ (c) involves an interactive session with the user. 

41 . (Currently Amended) The method of claim 33, wherein the building st^ (f) 

involves an active learning process. 

42. (Currently Amended) The computer implemented method of claims claim 33, 
wherein the communication criteria elicited in the eliciting st^ (c) is used, in part, by the 
system to label communications in the labeling st^ (d). 

43. (Currently Amended) The computer implemented method of claim 33, 
wherein the eliciting st^ (c) involves an interactive session with the user. 
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44. (Currently Amended) The method of claim 43, wherein the interactive session 
includes the steps of posing questions to the user regarding what type of information the user 
would consider relevant. 

45. (Original) The method of claim 44, wherein the interactive session also 
allows the user to provide keywords based upon a criteria the user considers relevant. 

46. (Canceled) 

47. (Original) The method of claim 44, wherein the questions elicit "yes", 
"no" and "unsure" responses from the user. 

48. (Currently Amended) The method of claim 43, wherein the building step (f) 
involves an active learning process. 

49. (Currently Amended) A computer memory containing a software program 
including instructions for implementing a method for developing a classifier for classifying 
electronic communications (text, electronic, etc.) comprising the steps of : 

(a) presenting communications to a user for labeling as relevant or irrelevant, the 
communications being selected from groups of communications including: 

a training set group of communications, the training set group of 
communications being selected by a traditional an active learning algorithm; 

a test set group of communications, test set group of communications for 
testing the accuracy of a current state of a ^ classifier being developed by the 
present method ; and 
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a previously-labeled set of communications previously labeled by at least one 
of the user, the system and another user; 

(b) developing die [[a]] classifier for classifying communications based upon the 
relevant/irrelevant labels assigned by the user during fee presenting step communications to 
the user; and 

(c) deploying the classifier for use in classifying electronic communications based 
upon the relevant/irreleyant labels . 

50. (Currently Amended) A computer memory containing a softv^^are program 
including instructions for implementing a method for developing a classifier for classifying 
electronic communications (text, electronic, etc.) comprising the steps of : 

(a) developing an expression of labeling criteria in an interactive session with the 

user; 

(b) presenting communications to a user for labeling as relevant or irrelevant; and 

(c) developing a classifier for classifying communications based upon the 
relevant/irrelevant labels assigned by the user during the presenting step ; and 

(d) deploying the classifier for use in classifying electronic communications based 
upon the relevant/irrelevant labels ; 

wherein at least one of the presenting step (b) and the developing step (c) use the 
expression of labeling criteria developed in the developing step (a). 
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5 1 . (Currently Amended) A computer system memory containing a software 
program including instructions for implementing a method for developing a classifier for 
classifying electronic communications (text, electronic, etc.) comprising the steps of : 

(a) defining a domain of communications on which a ^ classifier is going to operate; 

(b) collecting a set of communications from the domain; 

(c) eliciting labeling communication criteria from a user; 

(d) labeling, by the computer system, communications from the set of 
communications according, at least in part, to the labeling communication criteria elicited 
from the user; 

(e) labeling, by the user, communications from the set of communications; 

(f) building [[a]] the communications classifier according to a combination of labels 
applied to communications in labeling steps (d) and (e) ; and 

(g) deploying the classifier for use in classifying electronic communications based 
upon the combination of labels . 
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